Skip to content

Add Hyrise#883

Open
alexey-milovidov wants to merge 9 commits into
mainfrom
add-hyrise
Open

Add Hyrise#883
alexey-milovidov wants to merge 9 commits into
mainfrom
add-hyrise

Conversation

@alexey-milovidov

@alexey-milovidov alexey-milovidov commented May 8, 2026

Copy link
Copy Markdown
Member

Summary

  • Add a ClickBench entry for Hyrise, a research in-memory column-oriented DBMS from HPI.
  • Hyrise has no upstream binaries and needs gcc-15 / clang-20, so the build runs inside a multi-stage Dockerfile (ubuntu:25.04 builder → slim runtime image with hyriseServer, hyriseConsole, libhyrise_impl.so, libjemalloc.so.2). LTO is disabled by default: GCC's -flto=auto link spawns one lto1 per core and OOMs a 32 GB machine.
  • Load: hits.csv is split into 5M-row pieces which are imported sequentially (COPY ... FROM appends to an existing table), because Hyrise's CSV import reads the whole file into one std::string and materializes the table unencoded before dictionary-encoding it (~200 GB peak for the monolithic 75 GB file). Column types come from hits.csv.json next to each data file.
  • The first load exports the table to Hyrise's binary format via hyriseConsole (export bypasses the MVCC Validate operator, so the snapshot keeps the encoded segments; SQL COPY TO would materialize it unencoded). All cold-cycle reloads (BENCH_DURABLE=no) import that snapshot, which streams chunk-by-chunk and regenerates statistics.
  • Data size is reported from meta_segments.estimated_size_in_bytes since Hyrise has no on-disk persistence.
  • Hyrise's SQL is limited: no LENGTH, REGEXP_REPLACE, DATE_TRUNC, or OFFSET. ./query keys off psql's exit code so those queries (Q28, Q29, Q39–Q43) are recorded as null instead of stealing the timing line that psql still prints.

Closes #751

Test plan

  • Loaded a 1000-row CSV slice; SELECT COUNT(*), MIN/MAX(EventDate), AVG(UserID), COUNT(DISTINCT UserID), EXTRACT(MINUTE FROM EventTime) all return expected results
  • All 43 queries run end-to-end: 36 produce timings, 7 produce null (Q28/29 — unsupported functions, Q39–Q43 — OFFSET/DATE_TRUNC)
  • Output format matches ClickBench expectations: Load time: line, Data size: line, 43 lines of [t1,t2,t3],
  • Piecewise-load + binary-snapshot design verified against Hyrise master sources (csv_parser.cpp, import.cpp, binary_writer.cpp/binary_parser.cpp, console.cpp, sql_translator.cpp)
  • Full 100M-row run — needs a memory-rich machine (see the analysis in the comments); the estimated in-memory table size very likely exceeds c6a.4xlarge's 32 GB

🤖 Generated with Claude Code

alexey-milovidov and others added 5 commits May 8, 2026 20:56
Closes #751

Hyrise is a research in-memory column-oriented database from HPI
(https://github.com/hyrise/hyrise). It implements the PostgreSQL wire
protocol, so the benchmark connects via psql and uses Hyrise's
COPY ... WITH (FORMAT CSV) to load the standard ClickBench CSV dataset.

The system is built from source via Hyrise's install_dependencies.sh and
cmake/ninja; install_dependencies.sh requires Ubuntu 25.04 or newer.
Since Hyrise has no on-disk persistence, the data size is reported as the
total estimated segment size from the meta_segments meta table.

Hyrise has limited SQL coverage (no DATE/DATETIME types, no REGEXP_REPLACE,
no DATE_TRUNC). Queries that use unsupported functions are kept verbatim
and will be reported as null in the result file.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the build into a multi-stage Dockerfile (ubuntu:25.04 + gcc-15) so the
benchmark works on any Ubuntu host without polluting it with Hyrise's
toolchain. The runtime image only carries hyriseServer, libhyrise_impl.so,
libjemalloc.so, and a small set of shared-library deps (~250 MB).

Build args:
- HYRISE_REF (default master) — pin a Hyrise revision
- NO_LTO (default FALSE) — toggle LTO for faster development builds

Loading: drop create.sql and use hits.csv.json next to the data file as the
schema source. CREATE TABLE followed by COPY trips a Hyrise assertion
("set_immutable() should not be called on an empty chunk", chunk.cpp:125)
because COPY tries to seal the empty chunk left by CREATE TABLE; letting
COPY auto-create the table from the CSV meta avoids the issue.

run.sh: detect failed queries via psql's exit code rather than grepping the
output, so errors like "Invalid input error: Could not resolve function
'LENGTH'" are recorded as null. Hyrise lacks LENGTH, REGEXP_REPLACE,
DATE_TRUNC, and OFFSET, so 7 queries (Q28, Q29, Q39-Q43) are reported as
null; the remaining 36 succeed.

Tested locally on arm64: docker build produced a working image, hyriseServer
accepts psql connections, COPY loads a 1000-row sample, and run.sh produces
the expected 43 lines of [t1,t2,t3] output with nulls in the right slots.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split the old monolithic benchmark.sh + run.sh into the new layout:
install / start / stop / check / load / query / data-size, with
benchmark.sh now a thin shim that exec's lib/benchmark-common.sh.

BENCH_DURABLE=no because Hyrise has no on-disk persistence — the
driver re-runs ./load on every cold cycle and rolls that wall-clock
into the cold-try timing. The load script stages hits.csv (delivered
by download-hits-csv into cwd) under data/ next to hits.csv.json,
which the container exposes read-only at /data; the move is
idempotent so subsequent cold-cycle reloads reuse the staged file.

query keys off psql's exit code with ON_ERROR_STOP=1, so the seven
queries Hyrise can't run (Q28, Q29, Q39–Q43 — LENGTH, REGEXP_REPLACE,
DATE_TRUNC, OFFSET) propagate non-zero to the driver and get recorded
as null instead of stealing the Time line psql still prints for the
\timing meta-command.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bring in lib/benchmark-common.sh and lib/download-hits-* — the
hyrise/benchmark.sh shim added in f94c8af exec's ../lib/benchmark-common.sh,
but the branch had never been rebased onto main where lib/ lives, so the
shim died with "No such file or directory" on every cloud-init run.
@github-actions

Copy link
Copy Markdown
Contributor

The run of hyrise on c6a.4xlarge did not produce results.

Logs:

alexey-milovidov and others added 3 commits July 20, 2026 21:43
The benchmark run on c6a.4xlarge died linking `libhyrise_impl.so`:
GCC's `-flto=auto` spawns one `lto1` per core, and 16 of them exhaust
the machine's 32 GB RAM + 16 GB swap, so they get killed (`g++: fatal
error: Terminated signal terminated program lto1` — earlyoom's SIGTERM).
Hyrise's own CMake warns that LTO link times with GCC are very long, and
the runtime difference does not justify a build that cannot complete on
the reference machine. `NO_LTO=FALSE` stays available as a build arg.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`libhyrise_impl.so` links `libboost_container.so.1.83.0` and
`libboost_date_time.so.1.83.0` dynamically (visible in the CI link
command), but the runtime stage only installed boost-system and
boost-thread, which do not depend on them — `hyriseServer` would fail
to start with a missing shared library once the build succeeds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…shot

The monolithic `COPY hits FROM '/data/hits.csv'` cannot work at full
scale: Hyrise's CSV import reads the whole file into a single
std::string and materializes every column as unencoded ValueSegments
before dictionary-encoding them — roughly 200 GB peak for the 75 GB
hits.csv. `COPY ... FROM` into an existing table appends chunks, so
./load now splits the file into 5M-row pieces and imports them
sequentially, capping the transient overhead at one piece on top of the
growing encoded table.

The driver (BENCH_DURABLE=no) re-runs ./load on all 43 cold cycles, and
re-parsing the CSV each time would blow any time budget. The first load
therefore exports the table to Hyrise's binary format, which preserves
the encoded segments and streams chunk-by-chunk on import; every reload
imports that snapshot — the closest Hyrise equivalent of a durable
system re-reading its own on-disk format after a restart. The export
must go through `hyriseConsole`'s `export` command: server-side SQL
`COPY TO` wraps the table in an MVCC `Validate` operator and
BinaryWriter materializes the resulting ReferenceSegments as unencoded
value segments, so the snapshot would be raw-sized and would re-import
unencoded. The console is a separate process with its own catalog, so
it performs the piecewise CSV import itself and the server only ever
imports the snapshot (regenerating full-table statistics on add_table).

The runtime image now ships `hyriseConsole` (plus libreadline/
libncurses), and the /data bind mount is writable for the snapshot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member Author

Analyzed the failed run and pushed three fixes.

Why the run failed: the Docker build died linking libhyrise_impl.so — GCC's -flto=auto spawns one lto1 per core, and 16 of them exhausted the 32 GB + 16 GB swap of c6a.4xlarge (g++: fatal error: Terminated signal terminated program lto1 is earlyoom's SIGTERM). Fixed by defaulting the existing NO_LTO build arg to TRUE (ee02575).

Two more failures were waiting behind the build fix, both found by reading Hyrise's sources and fixed in this push:

  1. The runtime image was missing libboost-container1.83.0/libboost-date-time1.83.0, which libhyrise_impl.so links dynamically — hyriseServer would not have started (1bea24d).
  2. The monolithic COPY hits FROM '/data/hits.csv' cannot work at 100M rows: Hyrise's CSV import reads the entire file into one std::string and materializes the whole table as unencoded ValueSegments before encoding — roughly 200 GB peak for the 75 GB CSV. ./load now splits the CSV into 5M-row pieces and imports them sequentially (COPY ... FROM appends to an existing table), capping the transient overhead at ~13 GB above the growing encoded table. Since BENCH_DURABLE=no re-runs ./load on all 43 cold cycles, the first load also exports an encoded binary snapshot via hyriseConsole (its export command bypasses the MVCC Validate wrapper that would materialize the snapshot unencoded), and every reload streams that snapshot back instead of re-parsing 75 GB of CSV (8896065).

Machine-size caveat: even with the piecewise load, the peak is bounded below by the final dictionary-encoded in-memory table, which I estimate at 30–50 GB — most likely above what c6a.4xlarge (32 GB RAM + 16 GB swap) can hold. I'd suggest running this on a memory-rich machine via the manual workflow (e.g. c6a.metal); the c6a.4xlarge PR run will probably fail at load with an earlyoom kill, and if so the standard machine should get an {"error": ...} result.

🤖 Generated with Claude Code

alexey-milovidov added a commit to donge/ClickBench that referenced this pull request Jul 20, 2026
The PR benchmark always ran on c6a.4xlarge, but some systems cannot
work there — e.g. Hyrise (ClickHouse#883), an in-memory DBMS whose encoded hits
table alone exceeds the machine's 32 GB. The manual workflow can target
any machine, but its runs are invisible on the PR until the results
sink picks them up, and every re-run needs a manual dispatch.

A maintainer can now add one or more `machine:<ec2-type>` labels to a
PR (e.g. `machine:c6a.metal`) to override the default machine, one run
per label. Adding such a label relaunches the benchmark immediately;
other labels don't trigger anything. Labels can only be set by users
with triage access, and every launch stays gated by the
`benchmark-approval` environment, so the cost control is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

The run of hyrise on c6a.4xlarge did not produce results.

Logs:

@github-actions

Copy link
Copy Markdown
Contributor

Results for hyrise are ready for: c6a.metal.
The result files are committed as 330a464.

Logs:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Hyrise

1 participant